At study levels, it has been practiced to synthesize two or more face images, or at least one face image and one face graphics or a face animation to thereby create a fictional face image, although few of them are commercialized. With a conventional method, a large number of characteristic points, like three-dimensional polygon mapping 100 as shown in FIG. 1, are inputted over the face in each of two images to thereby synthesize the faces attached to a face standard model. This method requires extraction of a large number of characteristic points over the face and complicated calculation, which are difficult to execute on a portable terminal.
As a related technology, Japanese Laid-Open Patent Application (JP-P2003-242480A) discloses an image processing method and a server for utilize this method. This image processing method matches an existing image stored in an image database of a server with a portrait inputted into a user terminal to generate an intermediate image in accordance with user's customization requests and then provide it to the user terminal. That is, a face image is transmitted to the server through a network, synthesis processing is performed in the server, and the results are transmitted again to the terminal through the network.
Japanese Laid-Open Patent Application (JP-A-Heisei, 9-62865) discloses a face image processing method and a face image processing device. This face image processing method, using two face images (G1 and G2), a face image (G3) serving as a background, and shape models matched with these face images (G1, G2, and G3), respectively, synthesizes the face images with characteristic of the face images (G1 and G2) added to the face image (G3) serving as a base. Focusing on the characteristic portion of the face image (G1), by converting coordinates at a predetermined point corresponding to the characteristic portion of the shape model of this face image (G1) into coordinates at a corresponding point of the shape model of the background face image (G3), the face image (G2) is roughly matched with the background face image (G3), then coordinate values at points respectively corresponding to the post-matching shape models of the face images (G1 and G2) and the shape model of the face image (G3) are subjected to shape interpolation with a predetermined ratio, and then these shape models subjected to the shape interpolation are subjected to color interpolation with a predetermined ratio for respective corresponding pixels.
Japanese Laid-Open Patent Application (JP-A-Heisei, 10-91808) discloses a face image information conversion method and a face image information converter. This face image information conversion method includes the steps of: based on face image information including a plurality of different expressions of a predetermined human being, with the entire face serving as a processing unit, extracting a plurality of expression components for each of the expressions respectively corresponding to components of projection to a plurality of coordinate axes on a face image expression space; calculating a plurality of synthetic expression components corresponding to the face image information of the synthetic expression provided with an arbitrary ratio between the two arbitrarily specified different expressions, which is achieved by interpolation of the expression components corresponding to the two different expressions; and generating synthesized face image information in accordance with the plurality of synthetic expression components.
Japanese Laid-Open Patent Application (JP-P2004-178163A) discloses an image processing method and a device thereof. This image processing method includes the steps of: inalienably associating decoration and arrangement information of this decoration in a body part area; setting position of the body part area in an input image; by using the arrangement information associated with the decoration, defining the arrangement of the decoration so that it matches with the position of the set body part area and synthesizing the decoration with the input image; and outputting the synthesized input image.
Japanese Laid-Open Patent Application (JP-P2004-5265A) discloses an image synthesis method, an image synthesis device, and an image synthesis system. This image synthesis method includes: face image acquisition step for acquiring two face images, a base image generation step for generating a base image by synthesizing the two face images described above through morphing synthesis; a characteristic face image acquisition step for acquiring a characteristic face image; and a character addition step for generating a character-added image by synthesizing the characteristic face image described above and the base image described above through morphing synthesis.
Japanese Laid-Open Patent Application (JP-P2003-296713A) discloses a face image synthesis apparatus, a face image synthesis method and a program for executing this method, and a communication terminal provided with this face image synthesis apparatus and a communication method performed by this communication terminal. This face image synthesis apparatus is provided with: image input means adapted to take in an image including a human face; facial region acquiring means adapted to acquire a facial region including a head part of the human being from the taken image; and facial region processing means adapted to process image data in the acquired facial region to create a new face image.
Japanese Laid Open Patent Application (JP-P2000-67261A) discloses a face moving image creation method and device, and a recording medium on which this method is recorded. This face moving image creation method has: in a face moving image creation method in which two human face images whose eye and mouth open/close states are different from each other when viewed almost from the front are inputted to create a moving image, an eye position detection stage for detecting the position of the eyes based on the inputted two human face images; a mouth position detection stage for detecting the position of the mouth based on the inputted two human face images; an eye deformation control point arrangement stage for arranging around the eyes control points for deformation based on the eye position information detected in the eye position detection stage and associating the control points of the two human face image with each other; a mouth deformation control point arrangement stage for arranging around the mouths control points for deformation based on the mouth position information detected in the mouth position detection stage and associating the control points of the two human face image with each other; an in-between image generation stage for generating a plurality of in-between images with different eye and mouth open/close states obtained by deforming the inputted two human face images based on the control point information on the eyes and the mouth arranged in the eye deformation control point arrangement stage and the mouth deformation control point arrangement stage, respectively; and an image synthesis processing stage for synthesizing the human face moving images based on the plurality of in-between images generated in the in-between image generation stage and the two inputted human face images.
One of problems associated with synthesis of two face images is that simply performing weighted averaging while matching only one point may dislocate places, such as eyes or the like, whose color is largely different from that of the skin, so that these places remain like ghosts, thus resulting in unnatural synthesis. Thus, as described above, a method using three-dimensional polygon mapping is used. With this method, the outstanding characteristic parts on the two faces described above can be matched with each other. Thus, the ghost no longer appears, but it is required to extract a large number of characteristic points over the faces and perform complicated calculation, which are difficult to execute by a portable terminal with a limited input interface or limited calculation processing capability in particular.
There is another problem attributable to the nature of face synthesis. Facial characters include texture and arrangement of major parts like eyes, a nose and a mouth. For example, to mix together the face of a person A and the face of a person B with a ratio of 7:3, they need to be mixed for two purposes. That is, the mixing ratio needs to be 7:3 for the texture; and 70 percent needs to be set for the face of the human A and 30 percent needs to be set for the face of the human B in facial part arrangement. The facial parts arrangement mixture requires complicated calculation when a polygon mapping related method is applied. As a method of avoiding the problem involved in the practice with the portable terminal mentioned above, there is, as in Japanese Laid-Open Patent Application (JP-P2003-242480A), a method of transmitting face data to a server via a network and performing synthesis on the server, although many users have difficulty in accepting this method since the method involves communication charge and is also time-consuming depending on network conditions.